Script Datasets

Dataset focuses on running python scripts on the remote machine, however it is not locked into this run method.

Running with Script

Instead of setting the Dataset function to a python function, you are also able to use a Script object.

To enable this, you should first create a template.

Then, instead of using append_run with the arguments of a function, you can use the parameters that exist within your template.

Note

It is also possible to specify None as a function, and the arguments will be instead substituted into the Computer.script() method in addition to the run_args.

Output

Running in this manner means that this script will be executed and the stdout captured as the “result”. stderr is captured as normal.

To demonstrate this behaviour lets set up a run that exports an environment variable in the submission script, and then uses that parameter within the actual run.

To do this, we will use the Dataset, Script and Computer modules.

from remotemanager import Dataset, Script, Computer

First, lets create a submission script that exports a variable to be used:

template = 'export sub_value=#sub_value#'

url = Computer(template=template)

Now, lets use sub_value.

Here, we’re using the bash arithmetic wrapper to multiply two numbers together.

Note

Note that since our “result” has to be present on stdout, we should echo this value. As a result, it will always be a string.

template = "echo $(( sub_value * #inner_value# ))"

script = Script(template=template)

Looks good, lets set up the Dataset.

Passing the script as the function allows us to also pass the url explictly.

Additionally, we will set the sub_value parameter here.

ds = Dataset(script, url=url, skip=False, sub_value=10)

You should set up your dataset as normal, but explicitly pass None to the function argument

From here on out, it’s as usual:

for i in range(5):
    ds.append_run({"inner_value": i+1})

ds.run()

ds.wait(1, 10)

ds.fetch_results()

ds.results

appended run runner-0
appended run runner-1
appended run runner-2
appended run runner-3
appended run runner-4
Staging Dataset... Staged 5/5 Runners
Transferring for 5/5 Runners
Transferring 13 Files... Done
Remotely executing 5/5 Runners
Fetching results
Transferring 10 Files... Done

['10', '20', '30', '40', '50']

This output is exactly what we’d expect!

Again, note that the output is in string format, so must be converted if you require it in some other format.

Other Output Methods

Should you require other output formats, you are able to output them to a file. This file should then be registered as an extra_file_recv.